Skip to content

fix(examples): example 39 must run, and must name the error it really raises - #843

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:example-error-handling
Oct 2, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:example-error-handling

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

examples/39_error_handling.py has two defects, and neither is visible from CI: examples are not
executed by the test job, and no suite read this page's prose.

1. The file does not run. Its own docstring says "Every case below is executed and its real
behaviour reported". At upstream/main it stops at case 2:

   -- 2. laya.load on a path that does not exist --
Traceback (most recent call last):
  File ".../examples/39_error_handling.py", line 36, in <module>
    missing = os.path.join(os.path.dirname(MODELS["english"]), "does-not-exist")
                           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not tuple

MODELS is the Router spec dict, and examples/_common.py's checkpoint() returns the local
directory only when models/ exists in the checkout -- otherwise (BUNDLE_REPO, subfolder).
models/ is not in this repository, so every clone that runs this example gets the tuple. Cases 3,
4 and 5 -- the three that need an agent -- never ran anywhere.

The fix takes the path from LOCAL_MODELS, which is a directory by contract:

missing = os.path.join(LOCAL_MODELS["english"], "does-not-exist")

2. Item 3 describes an error that does not happen. The banner promised:

  1. a choice question with empty criteria -> a low-level RuntimeError, not an answer;

and under its own timings it printed:

both the empty dict and the empty list reach the scorer with zero options and blow up in top-k.

Neither is true, and the page printed the contradiction two lines above it:

   criteria={} -> ValueError: question 'q': a choice question needs at least one criterion
   criteria=[] -> ValueError: question 'q': a choice question needs at least one criterion

Agent._check_question refuses both shapes inside predict_batch before the state is encoded --
it never reaches render_options, the decision head, or top-k. Measured with the English
checkpoint, the traceback is predict -> system_one -> predict_batch -> _check_question.

Why

Case 3 is the shape a caller produces from a typo'd preset or an empty form field. A reader who
trusts the banner writes except RuntimeError, the ValueError escapes it, and the failure they
get is worse than the one documented. And the whole page -- the one an AGENTS.md points a
contributor at for "the failure modes a caller should handle" -- exits 1 on a fresh clone, so
nobody finds out.

tests/test_load_errors.py already owns cases 2 and 4 of this page (sections "missing directory"
and "a question whose options do not fit"), builds a loadable tiny checkpoint, and is in the CI
matrix. Section 7 spends that tiny agent on the page: it runs the example and lets the page's own
output be the witness.
For each numbered case, the exception types the banner lists must be
exactly the types that case's section prints. Rules:

  • The page is exec'd with _common.load answered by the tiny checkpoint built 40 lines above, so
    the crash class is caught by execution, not by regex. Its output is sliced per heading; the banner
    is outside those slices, so the banner cannot vouch for itself.
  • Router specs are witnessed as non-paths by calling _common.checkpoint("english") with
    has_local stubbed off -- the state of every CI job -- rather than asserting "it's a tuple", so a
    future str-shaped spec representation still passes while a real directory would not.
  • Case 3's conclusion must name _check_question: ValueError alone fits a hundred other raises.
  • The guard the page quotes (len(markers) != len(render_options(q))) is pinned to the comparison
    in laya/agent.py through a regex on the assignment, so the count variable can be renamed; the
    arm that renames n_opts -> opts_count stays green.
  • The option counts in its prose are the range() bounds in its code, and each "N options" label
    must name the dict that actually has N entries -- an arm that pairs "120 options" with the
    160-entry dict goes red, which is the trap of "fixing" this page by editing only the prose.
  • No sentence may send a rejected question to the decision head (top-k, reach the scorer,
    blow up). The two sentences this PR replaced are kept verbatim inside the test as controls, so a
    rule that matches nothing cannot pass for being empty.

How it was verified

python -B tests/test_load_errors.py          # 23 -> 49 checks, 49 passed 0 failed
python -B examples/39_error_handling.py      # runs to case 5 on the English checkpoint
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python -m compileall -q laya/ tests/

upstream/main's tests/test_load_errors.py counts 23 checks; here 49 (+26), exit 0. No laya/
file is changed by this PR. The fixed page, real English checkpoint, MPS, torch 2.14.0:

   -- 1. Router with an unknown model name --
   raised ValueError: unknown model 'bogus'; choose one of ['english', 'multilingual', 'typed-decisions'] (or an alias: [...])
   -> the name is normalised in the constructor, so a typo fails before any weights load.

   -- 2. laya.load on a path that does not exist --
   raised FileNotFoundError: Local model path not found: '.../models/laya/does-not-exist'. Check that the directory exists and that training saved the model successfully.
   -> an absolute or ./-relative path is treated as local and never hits the network.
   device: mps   (torch 2.14.0)

   -- 3. a choice question with an empty criteria dict --
   criteria={} -> ValueError: question 'q': a choice question needs at least one criterion
   criteria=[] -> ValueError: question 'q': a choice question needs at least one criterion
   -> neither form reaches the decision head. `Agent._check_question` runs inside
      `predict_batch`, before the state is encoded, and raises ValueError naming the
      question and what to add -- so an empty schema is a caller bug with a message that
      says which of twenty questions to fix, not a low-level crash to pattern-match.
   (a two-option question with None descriptions is fine: a=0.381 b=0.619)

   -- 4. options that do not fit the sequence --
   120 options -> NO exception; squeezed to 'opt_010' (confidence 0.0657)
   160 options -> raised ValueError: question 'q': only 126 of its 160 option markers fit in max_len=512 with head_max_len=192 spent on the question; lower head_max_len, raise max_len, or use fewer options
   -> the guard is `len(markers) != len(render_options(q))`: markers past max_len are
      dropped, so it fires only when whole option blocks fall out of the window. 120
      options slip through with a tiny per-label budget -- low accuracy, no warning.

   -- 5. a noul answer near 0.5 --
   single_problem   noul=0.5812  confidence=0.5812
   first_contact    noul=0.3223  confidence=0.6777
   wants_phone_call noul=0.1005  confidence=0.8995
   closest to 0.5: noul=0.5812 -> policy says ABSTAIN (needs more context or a human)
   -> a probability near 0.5 is not an error; it is the model honestly saying it cannot
      tell. Thresholding at 0.5 would silently turn that into a confident yes/no.

Cases 4 and 5 read the same as before because they were only ever wrong in the banner's framing of
case 3 -- the crash meant nobody had seen them since models/ stopped being vendored.

The gate was mutation-checked over 17 arms, each restoring every touched file from a byte snapshot
and requiring the suite's completion line (a crash or an aborted run scores ABORT, not a kill). All
17 arms:

OK       base                                                          failed=0 []
OK       M1 the crash at case 2 comes back                             killed by 4/4
OK       M2 banner item 3 promises a RuntimeError again                killed by 1/1
OK       M3 the top-k conclusion comes back                            killed by 2/2
OK       M4 the conclusion stays true but drops the validator's name   killed by 1/1
OK       M5 the runtime rejects the empty schema with a RuntimeError   killed by 1/1
OK       M6 the guard's option count is renamed                        failed=0 []
OK       M7 the guard compares the markers to the ceiling              killed by 1/1
OK       M8 the page paraphrases the guard instead of quoting it       killed by 1/1
OK       M9 the 120-option label is paired with the 160-option dict    killed by 1/1
OK       M10 the prose count drifts from the code count                killed by 1/1
OK       M11 case 2 is deleted rather than fixed                       killed by 2/2
OK       M12 case 3 stops printing the exception it caught             killed by 1/1
OK       M13 the first two banner lines swap their outcomes            killed by 2/2
OK       control: the conclusion is one print call, same words         failed=0 []
OK       control: the banner arrows are re-aligned                     failed=0 []
OK       control: the empty-criteria message is reworded               failed=0 []

17 arms, 0 not as expected

M5 mutates laya/agent.py (the raise becomes RuntimeError), so the gate is shown to be watching
the runtime and not merely the words -- and the banner then disagrees with the page's own output.
M11 is the arm that closes the cheap way out: deleting case 2 also makes the file run.

The first harness run was 15 of 17, and both misses were my gate, which is commit 3:

  • the os.path walk tested func.value.id == "os.path", which is never true for
    os.path.join(...) (the callee is Attribute(value=Attribute(Name('os')))), so the rule that
    refuses a Router spec to a filesystem call saw no call at all -- M1 went 3/4 instead of 4/4;
  • the option-count rule searched a space-joined copy of the page, so a sentence wrapped across two
    print calls never matched (\d+) options: the set it policed was empty and M10 survived.

Commit 3 fixes both (_dotted() walk, whitespace collapsed before the prose is searched) and adds a
non-vacuity check to each: the os.path rule asserts that at least one os.path call is visible,
and the count rule asserts the prose states a count.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes #

No issue: found by running the examples. No public API changed -- examples/39_error_handling.py
and tests/test_load_errors.py only, and the new checks live in the suite CI already runs
(python tests/test_load_errors.py).

…r it really raises

`os.path.dirname(MODELS["english"])` handed `os.path` a `Router` spec: `_common.checkpoint`
returns `(repo, subfolder)` when `models/` is not in the checkout, which is every checkout, since
`models/` is not in the repository. The file died with `TypeError: expected str, bytes or
os.PathLike object, not tuple` at case 2, so cases 3, 4 and 5 never ran under a docstring that
says every case is executed. The path now comes from `LOCAL_MODELS`, which is a directory by
contract.

Item 3 promised "a low-level RuntimeError" for a `choice` question with empty criteria, three
lines above the output that prints `ValueError`, and its conclusion sent both shapes to the
decision head: "reach the scorer with zero options and blow up in top-k". `_check_question`
rejects them before the state is encoded, naming the question -- measured with the English
checkpoint and with none.
Run the page on the tiny checkpoint this suite already builds, and let its own output be the
witness: for each numbered case, the exception types the banner lists must be exactly the types
that case's section prints. Nothing in CI executes examples, and no suite read this page's prose,
so both defects above were invisible.

Also pins the guard the page quotes to the comparison in `laya/agent.py` (by the name
`render_options` is stored under, so a rename survives), its option-count sentences to the
`range()` bounds it sets up, its missing-path case to `LOCAL_MODELS`, and no `os.path` call to a
`Router` spec -- with `_common.checkpoint` witnessed returning a non-path when `models/` is
absent. The two sentences this replaced are kept verbatim as controls, and three arms (renamed
guard variable, collapsed print, reworded message) are what the gate must survive.
Found by mutation, not by reading: the `os.path` walk matched `func.value.id == "os.path"`, which
is never true for `os.path.join(...)` -- the callee is `Attribute(value=Attribute(Name('os')))`. The
rule that is supposed to refuse a `Router` spec to a filesystem call therefore saw no call at all,
and the arm that reintroduces the crash only went three of four checks red. A dotted-name walk fixes
it, with a non-vacuity check that at least one `os.path` call is visible.

The option-count rule read a space-joined copy of the page, so a sentence wrapped across two `print`
calls never matched `(\d+) options` -- the count it was policing was the empty set, and an arm that
moves 120 to 110 survived. Whitespace is now collapsed before the prose is searched, and the rule
asserts the prose states a count rather than quietly passing on none.
@NandhaKishorM
NandhaKishorM merged commit 47a3dbd into NandhaKishorM:main Oct 2, 2026
23 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.24.

Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, compileall clean, both TypeScript packages green (laya-ts 597 tests, the SDK 26), and a decision diff against 0.3.23 on the real english and multilingual checkpoints (74 answers over choice, score and noul; short and long states; budgeted calls and predict_long) showing zero decision flips and a max probability delta of 0.000000.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants